HPAVC

home *** CD-ROM | disk | FTP | other *** search

/ HPAVC / HPAVC CD-ROM.iso / pc / HPACK78S.ZIP / docs / hpackstd.txt < prev next >

Wrap

Text File | 1992-12-03 | 63.8 KB | 1,438 lines

HPACK Technical Information =========================== (HPACK Level 1 Format) HPACKSTD.TXT Updated 1 July 1992 "Programming is like sex: One mistake and you have to support it for the rest of your life" The HPACK Philosophy: --------------------- HPACK was designed along four principles: - Reasonable compression. HPACK's compression currently outperforms all other known archivers running in comparable amounts of memory. - Archive flexibility and portability. HPACK's method of storing system-specific information was designed to be the most easily portable possible. The HPACK format can store virtually anything you care to throw at it. - Ease of use. The general rule was to only include the functionality that could be fitted on a single page of help screen. - The use of sound algorithms. Adding encryption and authentication options which people will trust their data to is of little use if they can be broken in a matter of moments. The following document describes the strategy used to implement these principles. This document describes the layout of the Level 1 HPACK format, which is a reduced version of the full HPACK format to enable it to run on current low-end systems (mainly memory-limited MSDOS machines). The Level 2 HPACK format consists of a superset of the Level 1 format which is specified entirely using ASN.1 (Abstract Syntax Notation One) and encoded using the Basic/Distinguished Encoding Rules. The Level 2 format is currently unimplemented due to its need for support for things like 64 and 128-bit data types, and the fact that it will be written entirely based on C++ STREAM objects allowing any program to easily access HPACK data files. In general at the time of writing those systems which have a usable C++ compiler are too limited to handle the Level 2 format, and those which can handle it don't have a C++ compiler worth using (or don't have a C++ compiler at all). Archive Layout: --------------- A standard HPACK archive is laid out as follows: ====================================== BYTE[4]: Archive ID: 'HPAK' ====================================== File 1 data -------------------------------------- . . . -------------------------------------- File n data ====================================== Dir 1 data -------------------------------------- . . . -------------------------------------- Dir n data ====================================== Dir 1 header . . . Dir n header -------------------------------------- File 1 header . . . File n header -------------------------------------- Dirname 1, '\0' . . . Dirname n, '\0' -------------------------------------- Filename 1, '\0' . . . Filename n, '\0' -------------------------------------- WORD: No directory headers WORD: No file headers LONG: Length of directory block WORD: Directory block checksum BYTE: Special info BYTE[4]: Archive ID: 'HPAK' ====================================== Each archive begins with the ID string 'HPAK', followed by the actual data stored in the archive. These files are stored as one continuous block of raw data, with an optional auxDataLength data block following each block of archived data. The file and directory headers are in a format described below. The fileNames and dirNames are stored in null-terminated ASCII or EBCDIC (depending on the source system) format with the extensions described below. The trailer information is stored as follows: typedef struct { WORD noDirHdrs; /* No.of directory headers */ WORD noFileHdrs; /* No.of file headers */ LONG dirInfoSize; /* Length of archive directory block */ BYTE specialInfo; /* The kludge byte */ WORD checksum; /* CRC16 of dir.block */ BYTE archiveID[ 4 ]; /* 'HPAK' or 0x4850414B */ } ARCHIVE_TRAILER; The dirInfoSize block covers the combined length of the dirHdrs, fileHdrs, and file and directory names. The checksum covers the dirInfoSize data as well as the fields in the trailer struct up to the checksum field itself (in other words all directory information apart from itself and the archiveID). The high three bits of the byte of special info currently have the following values for the prototype versions of HPACK: 0x20 - LZW version, used for development only. 0x40 - LZA version. Compression = 2.82 bits/byte (no longer used). 0x60 - LZA' version. Compression = 2.61 bits/byte. 0x80 - LZA" version. Compression = ~2.58 bits/byte (est). 0xA0 - MBWA prot.1. Compression = ~2.75 bits/byte (est). 0xC0 - MBWA prot.2. Compression = ~2.55 bits/byte (est). A special info ID with any of the 3 high bits set indicates the archive was created with a prototype version of HPACK. The release version of HPACK will use this byte as a bitfield to indicate various extensions to the standard format. Currently only the low 5 bits are used, the remaining high bits being used to indicate the prototype level: typedef struct { unsigned multiEnd : 1; /* Last part of multipart archive */ unsigned multipart : 1; /* Part of a multipart archive */ unsigned secured : 1; /* Secured archive */ unsigned encrypted : 1; /* Encrypted archive */ unsigned blockCopr : 1; /* Block compressed archive */ unsigned spare : 3; /* Currently unused */ } SPECIALINFO; There is also a possibility that one bit will be used for to flag the use of the notorious 'B' format (in which the archive directory as well as the archive data is compressed). Archive Directory Structure: ---------------------------- An HPACK file header is defined in FILEHDR.H, and is arranged as follows: First, there is a bitfield to hold information such as the system the archive was created on and various information pertaining to the file. This bitfield is described as a struct, but is handled internally via bitmasks to eliminate any endianness problems: typedef struct { unsigned origLen : 1; /* fileLen field WORD/LONG bit */ unsigned coprLen : 1; /* dataLen field WORD/LONG bit */ unsigned otherLen : 2; /* dirIndex/extraLen field length */ unsigned isSpecial : 1; /* Whether this is a special entry */ unsigned hasExtra : 1; /* If there is extra data attached */ unsigned isText : 1; /* Whether this is a text file */ unsigned dataFormat : 3; /* Storage type */ unsigned systemType : 6; /* The sys.the arc.was created on */ } ARCHIVEINFO; The data formats are as follows: enum { STORED, PACKED, <5 x spare> }; If the isSpecial bit is set then this entry is to be treated as a non-data file entry. The file header as stored on disk is followed by an additional WORD which indicates the type of the file. All files of type SPECIAL reside in their own namespace, making it possible to have multiple files of the same name in one directory provided their types are different. If the hasExtra bit is set, the header is followed by a BYTE with the following layout: typedef struct { unsigned secured : 1; /* Data is secured */ unsigned encrypted : 1; /* Data is encrypted */ unsigned linked : 1; /* File is linked to other file(s) */ unsigned unicode : 1; /* Filename is Unicode */ unsigned extraLength : 4; /* Length of extra information */ } EXTRAINFO; If the file is linked to another file, the link ID is stored in a WORD immediately following the header itself, but after the type SPECIAL word if there is one present. All files with the same link ID are seperate file entities which share common data and in some cases common attributes. The linkID is simply a magic number connecting all files with the same ID. Linked files or directories can be of two types, anchor nodes or dependant nodes. An anchor node contains the physical data for the linked file(s)/directorie(s). A dependant node is a file entry which points to an anchor node. Each link chain must have one or more anchor node (containing the actual file/directory data). A link ID has the following layout: typedef struct { unsigned anchor : 1; /* Whether this is an anchor node */ unsigned linkID : 15; /* Magic number for ID */ } LINKINFO; The extra information field is a deliberately short field used to store mission-critical information such as the following: Apple IIgs: WORD - File type LONG - Auxiliary file type Archimedes: WORD - File type Macintosh: LONG - File type LONG - File creator/owner The extra information is stored immediately following the header itself, but after the type SPECIAL word and link ID if there are any present. If only the encryption, authentication, or link bits are used, the extraLength field is set to 0. This field also allows for the storage of very short items of data as pure headers of type SPECIAL only (ie with no associated data). Currently this option is unused but may be used in the future. Data integrity checking is performed either by a CRC16 checksum on stored data or a checksum calculated by the LZA/MBWA compressor as part of the compression/decompression process. The CRC16's are only used on very short stored files (where their probability of picking up errors is good); for longer files the checksum mechanism which is built into the compressor itself is used. Taking into account the types of errors that will occur over transimission lines, a 16-bit CRC will catch all single bit errors, all two bit errors, all odd numbers of errors and all burst errors of up to 16 bits. The checksumming built into the compressor will by its nature detect virtually all errors (the exact details are quite complex to explain and are based on the way the compressed data is encoded). The system types are as follows: enum { MSDOS, UNIX, AMIGA, MACINTOSH, ARCHIMEDES, OS2, IIGS, ATARI, VMS, PRIMOS, <53x spare> }; The file headers are stored as variable-length records, with the lengths of the fileLen, dataLen, auxDataLen, and dirIndex fields being given by various bits in the archiveInfo field. The first field written is the archiveInfo field itself; when the record is read in, the length bits are used to determine the lengths of the fields and the appropriate routines called to read in BYTE's, WORD's, or LONG's. These routines are independant of data endianness. The length bits are as follows: origLen: 0 = fileLen = WORD 1 = fileLen = LONG coprLen: 0 = dataLen = WORD 1 = dataLen = LONG otherLen: 00 = dirIndex = ROOT_DIR, auxDataLen = BYTE 01 = dirIndex = BYTE, auxDataLen = BYTE 10 = dirIndex = BYTE, auxDataLen = WORD 11 = dirIndex = WORD, auxDataLen = LONG The file header as stored in memory is as follows: typedef struct { LONG fileTime; /* Mod. date + time of the file */ LONG fileLen; /* Original data length */ LONG dataLen; /* Compressed data length */ LONG auxDataLen; /* Auxiliary data field */ WORD dirIndex; /* Where the file is in the dir.tree */ ARCHIVEINFO archiveInfo; /* File-specific bitflags */ } FILEHDR; The dataLen field is the length of the compressed data, not counting any extra information which may be stored in the auxDataLen field. The date field contains the modification date of the file, and is stored as the number of seconds since 00:00:00 GMT on 01/01/1970 (the time may not be entirely correct on systems which don't store time zone information). An HPACK directory header is defined in FILEHDR.H, and is arranged as follows: First, there is a bitfield to hold information such as the system the archive was created on and various information pertaining to the file. This bitfield is described as a struct, but is handled internally via bitmasks to eliminate any endianness problems: typedef struct { unsigned otherLen : 2; /* parentIndex/dataLen field length */ unsigned isSpecial : 1; /* Whether this is a special entry */ unsigned linked : 1; /* Dir is linked to other dir(s) */ unsigned unicode : 1; /* Directory name is Unicode */ unsigned spare : 3; /* Reserved */ } DIRINFO; The directory headers are stored as variable-length records, with the lengths of the dataLen and parentIndex fields being given by various bits in the dirInfo field. The first field written is the dirInfo field itself; when the record is read in, the length bits are used to determine the lengths of the fields and the appropriate routines called to read in BYTE's, WORD's, or LONG's. These routines are independant of data endianness. The length bits are as follows: otherLen: 00 = parentIndex = ROOT_DIR, dataLen = 0 01 = parentIndex = BYTE, dataLen = BYTE 10 = parentIndex = BYTE, dataLen = WORD 11 = parentIndex = WORD, dataLen = LONG The directory header as stored in memory is as follows: typedef struct { LONG dirTime; /* Mod. date + time of the directory */ LONG dataLen; /* The length of the sys-specific dir.info */ WORD parentIndex; /* The pointer to this directories parent */ BYTE dirInfo; /* Directory-specific bitflags */ } DIRHDR; The dataLen LONG defines the length of the directory data field, which contains system-specific information such as permission bits for the directory. The date field contains the modification date and time of the directory in the same format as the file date. If the isSpecial bit is set, it indicates that this entry is to be treated as a special directory entry. The directory header as stored on disk is followed by an additional WORD which indicates the type of the directory. If the directory is linked to another directory, the link ID is stored in a WORD immediately following the header itself, but after the type SPECIAL word if there is one present. All directories with the same link ID are seperate directory entities which share common data and in some cases common attributes. The linkID is simply a magic number connecting all directories with the same ID, and has the same format as file linkID's. The layout of file and directory headers as stored within an archive is shown below. All the variable or optional fields are controlled by the flag bits outlined above. File Header: WORD : Header information bits which control the following fields. None or BYTE or WORD : Index of directory file is in None or BYTE or WORD or LONG : Auxiliary data size LONG : File time WORD or LONG : Original file size WORD or LONG : Compressed file size Optional WORD : File type Optional BYTE : Extra header information bits which control the following fields. Optional WORD : Link ID Optional variable-length field : System-specific extra information. Directory Header: BYTE : Header information bits which control the following fields None or BYTE or WORD : Index of directory directory is in None or BYTE or WORD or LONG : Directory data size LONG : Directory time Optional WORD : Directory type Optional WORD : Link ID File and directory names are stored as null-terminated ASCII or EBCDIC strings, depending on the system they originated on. In addition, HPACK will recognise Unicode/ISO 10646 strings as file and directory names as well as standard ASCII/ISO 646 or EBCDIC ones. Unicode is used since it encompasses virtually all of the widely-varying standards for non-ASCII fonts and character systems available today. The multibyte character set employed in Unicode is stored externally in big-endian format as is the rest of the data used in HPACK. HPACK will recognise Unicode data in file and directory names but currently ignores it. HPACK Directory Structure: -------------------------- Unlike many other archivers, HPACK doesn't store directories merely as an extension to a filename. Instead, HPACK maintains a full internal directory tree allowing comlete reproduction of any directory structure. The internal directory structure which is laid out as follows (assuming the following sample directory structure): / -+- A -+- B -+- C This directory tree is encoded in a manner similar | | | to that of the strings in LZW compression. Every | | +- C1 entry contains a pointer back to a chain of its | +- B1 parents. Directories are scanned in depth-first +- A1 -- B2 order. The tree is stored in a dynamically-sized | array which also contains a list of files and sub- +- A2 directories for each directory, and an index for the directory name in the string table. A sample run on the above directory is: +-------------------------------+ +----+ +----+ | v | v | | A = 1, B = 2, C = 3, Pop to B (pred(C)), C1 = 4, Pop to B (pred(C1)), +------------+ +------------+ +----+ v | v | v | Pop to A (pred(B)), B1 = 5, Pop to A (pred(B1), Pop to / (pred(A)), A1 = 6, B2 +-------------+ v | Pop to A1 (pred(B2)), Pop to / (pred(A1)), A2 = 8, Pop to / (pred(A2)), Exit. Typical links within the array are shown below: +-----------------+ +-----|-------------+ | +-----|-+ +-----+ | | +---+ | | +-+ | | | v | v | v | | | | +---+|--+|--+|--+|--+|--+|--+-/ /--+---+ | | | | | | | | / / | | +---+---+---+---+---+---+---+-/ /--+---+ The root directory is directory 0 and is never explicitly written to or read from disk; all subdirectories are added off the root directory, starting with a directory index of 1. The directory structure should be stored in depth-first order, and with contiguous directory numbers; there is code in ARCDIR.C to rebuild the directory tree in the correct order which should be called after changes have been made to the directory structure. Storing Miscellaneous Data: ---------------------------- All crufties in an HPACK archive are stored in a files' auxData field and a directories' data field in a tagged format. Each file and directory can have up to 4GB of crufties associated with it, typically attributes, icons, access control information, GUI information such as window positions and sizes, and so on. The tags used to denote the crufties are either 16-bit or longer variable-length values, with the following format: Short-format tags: These tags are of the form <10 bits Tag ID><6 bits data length>. Tag ID's range from 0x000-0x3FE. typedef struct { unsigned tagID : 10; /* Tag ID for this tag */ unsigned tagLen : 6; /* Length of this tag */ } SHORT_TAG; Long-format tags: These tags have a short-format tag ID of 0x3FF, and are treated as one 24-bit value by combining the tag ID and data length fields, and the following byte. The actual data length is given in a BYTE, WORD or LONG which follows the basic tag. If the tag data is compressed, the uncompressed data length is given in a WORD or LONG following the actual length. The size of the length fields is indicated by the dataLength bits, and are as follows: dataLen = 00 length = BYTE, uncompr.length = WORD dataLen = 01 length = WORD, uncompr.length = WORD dataLen = 10 length = WORD, uncompr.length = LONG dataLen = 11 length = LONG, uncompr.length = LONG typdef struct { unsigned longTagID : 10; /* Always 0x3FF for long-format tag */ unsigned dataLen : 2; /* Length of length/ucoprLen fields */ unsigned dataFormat: 3; /* Storage type */ unsigned tagID : 9; /* Tag ID for this tag */ } LONG_TAG; This format results in 1023 short-form tags and 512 long-form tags. Tag length fields cover the tagged data itself, but not any parts of the tag header. Example of short format tag: MSDOS attributes (1 byte): WORD : ( MSDOS_ATTR | 1 ) [data: 1 byte] Example of uncompressed long format tag: Authentication information (84 bytes): WORD : ( LONG_BASE | LONGTAG_BYTE_WORD | TAGFORMAT_STORED ) BYTE : SECURITY_INFO BYTE : 0x54 [data: 84 bytes of authentication information] Example of compressed (packed) long format tag: Long format comment (182 bytes, compressed to 92 bytes): WORD : ( LONG_BASE | LONGTAG_BYTE_WORD | TAGFORMAT_PACKED ) BYTE : COMMENT BYTE : 0x5C BYTE : 0xB6 [data: 92 bytes of compressed comment] Example of compressed (packed) extended tag: Macintosh resource fork (269,128 bytes, compressed to 167,241 bytes): WORD : ( LONG_BASE | LONGTAG_LONG_LONG | TAGFORMAT_PACKED ) BYTE : RESOURCE_FORK LONG : 0x00041B48 LONG : 0x00028D49 [data: 167,241 bytes of compressed resource fork) All tags are defined in the file TAGS.H, which acts as a master tags record for all versions of HPACK. If an unknown tag is encountered, the tags length field may be used to skip the information which it defines. Note that HPACK has routines called writeTag() and readTag() which automatically sort out what types of tags to use and how to store information on lengths etc. The above information is provided mainly to give an indication on how the data is actually stored within an archive. Encryption/Authentication Data Format: -------------------------------------- The format for encryption/authentication information used by HPACK is vaguely compatible with that used by Philip Zimmerman's PGP encryption package. HPACK recognises three types of security information packets: Signature packets: A signature packet contains the algorithm ID of the signing algorithm, the key ID of the key used to create the packet, and the message digest packet stored as an encrypted multiprecision integer. Algo CTB len ID keyID Encrypted MPI - MD packet crc16 +-+ +-+ +-+ +-+-+-+-+-+-+-+-+ +-------------------------+ +-+-+ | | | | | | | | | | | | +-+ +-+ +-+ +-+-+-+-+-+-+-+-+ +-------------------------+ +-+-+ A message digest information packet contains the algorithm ID of the message digest algorithm, the message digest itself, and a timestamp stored in the usual format. Algo CTB len ID Message digest Timestamp +-+ +-+ +-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+ | | | | | | | | | | +-+ +-+ +-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+ A message digest packet may be prepended to signed data and contains the algorithm ID of the message digest algorithm. Algo CTB len ID crc16 +-+ +-+ +-+ +-+-+ | | | | | | | | +-+ +-+ +-+ +-+-+ Conventional-key encryption packets: A conventional-key encryption packet contains the algorithm ID of the encryption algorithm and any extra keying information needed by the algorithm. This keying information usually takes the form of a 64-bit IV used by the block cipher employed for the conventional-key encryption. Algo Keying CTB len ID information crc16 +-+ +-+ +-+ +--------+ +-+-+ | | | | | | | | | | +-+ +-+ +-+ +--------+ +-+-+ Public-key encryption packets: A public-key encryption packet contains the algorithm ID of the public-key encryption algorithm, the key ID of the key used to create the packet, and the conventional key encryption information stored as an encrypted multiprecision integer. Use of an IV is unnecessary since the conventional-key encryption information contains a random key which differs for each packet. Algo CTB len ID keyID Encrypted MPI - CK encr.info crc16 +-+ +-+ +-+ +-+-+-+-+-+-+-+-+ +----------------------------+ +-+-+ | | | | | | | | | | | | +-+ +-+ +-+ +-+-+-+-+-+-+-+-+ +----------------------------+ +-+-+ A conventional key encryption information (DEK) packet contains the algorithm ID of the encryption algorithm, and the key material needed by the algorithm (this varies from algorithm to algorithm). Algo CTB len ID Key material +-+ +-+ +-+ +-------------------------+ | | | | | | | | +-+ +-+ +-+ +-------------------------+ Multiprecision integers are stored in big-endian format with a 16-bit prefix which gives the number of significant bits. If the bitcount is not a multiple of 8, the remaining bits are zero-padded. A multiprecision integer with a value of zero is stored with a zero in the bitcount field and no following value byte. The CTB is the cipher type byte, which specifies the type of data structure which follows it. The CTB bits have the following meaning: Bits Content 7-4 1010 - Designates the byte as a CTB 3 'More' bit - if set indicates another packet follows 2-0 Packet type field: 000 - Public-key-encryption packet 001 - Signature packet 010 - Message digest packet 011 - Message digest information packet 100 - Conventional-key-encryption packet 101 - Conventional-key-encryption information (DEK) packet 110 - Reserved for future use 111 - Reserved for future use Data authentication information is given by way of a signature packet. Encryption information is given either by way of a conventional- or public-key encryption packet. Archive Encryption: ------------------- HPACK archives can be block encrypted with either public- or private-key encryption schemes. The archive directory and archive data are encrypted seperately, allowing two levels of access to the archive, either to information on the contents of the archive (in terms of directory information), or full access to the entire archive. If the archive is block encrypted, the archive data and directory information areas are prefixed with either a public- or conventional-key encryption packet containing the information necessary to allow HPACK to decrypt the following data. In addition the byte of special info will have the encrypted bit set to indicate that this is an encrypted archive. The archive data which is encrypted begins after the initial 'HPAK' ID string and ends at the start of the directory information; the directory data which is encrypted begins at the start of the directory information and ends at the start of the archive trailer. Thus the entire archive is enclosed in a security veil which allows no access to any information without the appropriate key(s) or password(s) to unlock it. Archive Authentication: ----------------------- HPACK archives can have authentication information attached to them, in which case the archive layout ends as follows: . . . Filename n, '\0' -------------------------------------- WORD: No directory headers WORD: No file headers LONG: Length of directory block WORD: Directory block checksum -------------------------------------- Authentication information -------------------------------------- WORD: Authentication info.length BYTE: Special info BYTE[4]: Archive ID: 'HPAK' ====================================== An extra block of data is inserted between the directory block checksum and the archive ID, consisting of a WORD containing the length of the authentication information, and the authentication information itself. In addition the byte of special info will have the secured bit set to indicate that this is a secured archive. The data which is checked begins after the initial 'HPAK' ID string and ends at the start of the authentication information itself; thus the entire archive is enclosed in a security wrapper which it is computationally infeasible to destroy. The authentication information can either be used to validate the archive, or it can be skipped and the archive handled as normal. If the archive is altered, a warning should be issued that this will destroy the authentication information. Currently authentication is handled by generating an RSA Data Security Inc. MD5 message digest for the archive and signing it with the RSA public-key cryptosystem. However, alternative forms of authentication (such as the use of the ElGamal PKC) can easily be substituted. Multipart Archives: ------------------- HPACK supports multipart archives by treating them as a single virtual archive, with the I/O code taking care of disk swapping requirements and so on. The main archiver code sees all archives as a single contiguous block of data, whether they are spread over several disks or not (it is for example possible to create a multipart archive without even using HPACK simply by splitting up an existing archive into segments and adding the archive ID to the start and the ID and multipart information to the end of each segment). The end of each internal segment is as follows: . . . Raw data -------------------------------------- WORD: Segment number BYTE: Special info BYTE[4]: Archive ID: 'HPAK' ====================================== The end of the final segment is as follows: . . . Filename n, '\0' -------------------------------------- WORD: No.directory headers WORD: No.file headers LONG: Length of directory block WORD: Directory block checksum -------------------------------------- Authentication information (if present) ====================================== LONG: Length of 1st segment . . . LONG: Length of nth segment -------------------------------------- WORD: Authentication info length WORD: Total number of segments LONG: Pos.of end of last segment WORD: Segment block checksum BYTE: Special info BYTE[4]: Archive ID: 'HPAK' ====================================== The segment list covers the entire archive to the start of the segment list itself. Unlike the rest of the archive, the segment list and final trailer are always located on the same disk; since no segmentation information is present when these are read in they must be on the same physical medium. If the segment list and trailer must be placed on a seperate disk, the high bit of the total segment count should be set to indicate that the rest of the archive is on a seperate disk. The segment block checksum covers all information from the start of the segment list to the checksum itself. When HPACK detects a full disk it should write as much of the current data as can still fit on the disk, append the trailer information, and request a new disk. The existing code does this by checking how much data was written and comparing it to the amount that was supposed to be written: If the result is less, the disk is full. A simpler approach would be to check how much room is available on the disk before writing it, however this will not work on multitasking OS's since the disk space can change between the call to determine the space and the call to write the data. The code in HPACK uses the data write to lock the disk space available to it, and then backtracks over the data if necessary. This atomic write ensures there can be no problems with multiple processes accessing the same disk volume. Reading a multipart archive is performed as a multi-stage process in which a bootstrap read is used to read the segments, and then the full archive is read as usual: Step 0: Locate end of archive Read in disk number and offset of segment list start Step 1: Ask for correct disk if not there already Seek to segment list start Step 2: Read in segmentation information Enable virtual filesystem I/O once segmentation information is available. HPACK Command Format: --------------------- It is recommended that the following commands be used for the CLI versions of HPACK. All CLI versions should be standardised to use this command format (where the commands are applicable) to make it possible to use HPACK on any system without having to relearn the command set. All system-specific switches should be specified using the -z option (eg under DOS -zv = use volume label, -zs = use disk serial number, under Unix zlower = force lowercase on file/directory names, znoumask = ignore umask on file/directory creation). See HPACK.DOC for more information on these commands. Command letters are: [A] - Add files to an archive. [X] - Extract files from an archive. [V] - Directory of files inside an archive. [P] - View a file within an archive. [T] - Test the integrity of an archive. [D] - Delete files from an archive. [F] - Freshen files to an archive. [R] - Replace files in an archive. [U] - Update files to an archive. Option letters are: -0 - Store without compression. -a - Store file attributes. -b - Specify a base pathname for files. -c - Encrypt files (public- or private-key encryption). -d - Directory options (Mkdir, Rmdir, Mvdir, path specifiers etc). -e - Add error recovery information. -f - Force file move. -i - Interactive mode - prompt for all files. -k - Overwrite existing archives. -l - Add authentication information. -m - Generate multipart archive -o - Overwrite on extraction options (-oa All, -on None, -os Smart, -op Prompt) -r - Recurse through subdirectories. -s - Stealth mode. -t - Touch files on extraction. -u - Unified compression mode. -v - View files options (-vf Files, -vd Directories, -vs Sort files). -w - Treat files as archive comments. -x - Translate options for text files (-x smart, -xr CR, -xl LF, -xc CRLF, -xxnn Hex, -xe EBCDIC, -xp Prime, -xa ASCII as appropriate) -z - System-specific commands If possible the following wildcard chars should be used (these are the Unix chars which seem to be the most common ones): * - Matches multiple characters ? - Matches any one character [...] - Matches any of the enclosed range of characters '...'. [^...] - As above, but matches anything *not* in the range. \ - Literal char escape (this becomes '#' in the Atari ST, MSDOS, and version since these systems use '\' as the directory seperator). HPACK Program Structure: ------------------------ The HPACK archiver has a number of layers which interface between the user and the host OS. This archiver structure is shown below, with the user interface at the top and the filesystem interface at the bottom. The core archiver routines, which should need few changes, are in the centre. Note the multiple levels of filesystem I/O handling present towards the bottom - any necessary extra functionality for fileysystem I/O can be transparently added at this level. Host OS screen I/O ----------------------------------------------------- / \ / \ | | | | = data \ / \ / flow +---------------------------------------------------+ | GUI.C / CLI.C archiver frontend | ^ | FRONTEND.C, SCRIPT.C, DISPLAY.C user interface | | = control +---------------------------------------------------+ v flow ^ ^ ^ | | | v v v +------------+ +-----------+ +----------------------+ +------------+ | ARCDIR.C | | TAGS.C | | ARCHIVE.C | | ERROR.C | | ARCDIRIO.C | | | | compression manager | | error | | | | extra | +----------------------+ | handling | | archive | | data/ | | +------------+ | directory | |information| +----------------------+ +------------+ | manager | | manager | | Plug-in copr.modules | | HPAKTEXT.C | +------------+ +-----------+ +----------------------+ | language- | / \ / \ / \ | independant| | | | | | | | text system| \ / \ / \ / +------------+ +---------------------------------------------------+ | CRYPT.C encryption/authentication handling | +---------------------------------------------------+ / \ | | \ / +---------------------------------------------------+ | FASTIO.C, FILESYS.C virtual file I/O | +---------------------------------------------------+ / \ | | \ / ----------------------------------------------------- Host OS file I/O HPACK and Portability: ---------------------- HPACK has been written to be as portable as possible - the main difference between the CLI and GUI versions are the use of either cli.c or gui.c at the highest level of the user-interface system in the above diagram. All other code is common to both CLI and GUI versions for all operating systems. The baseline version of HPACK is the original CLI version (which has been around for about two years longer than any other version). The portability problems, at least to another CLI-based OS, are apparently not severe: The Xenix port of version 0.71 of HPACK (by Stuart the Hut) took only a few days to accomplish. The original Macintosh port of version 0.77 was done in a single day. Level 1 HPACK has been successfully compiled using the following compilers: - DICE - Generic Unix cc (many variants) - Irix cc - Microsoft C - MiNT gcc - Norcroft Arm C - Orca/C - SAS/Lattice C - SunOs acc - Think C - TopSpeed C - Turbo C - Unix gcc - VAX vcc - Watcom C (the only C compiler named after a toilet) - Zortech C Getting Started: ---------------- "Any given HPACK port, when running correctly, is obsolete" If your source came in a zipfile, unzip it with the -d option to create subdirectories as appropriate. If it came as a tar.Z the files will already be in the correct directories. Edit DEFS.H for your system, and edit the supplied makefile for whatever your setup is. If your system supports system-level I/O most of the files should compile without too many errors, especially if the system is at least vaguely Unix-compatible. The nonstandard functions are aliased to calls to hputs() which print a warning message that they should be implemented later. For a CLI-based version all that is left is to implement the nonstandard functions and tune things like the handling of paths and filenames for each system. Most of this is contained in FILESYS.C and <os_name>.C One point to note is that the MD5Transform() routine in MD5.C seems to break some optimizing compilers, so the version which has been broken up into four parts may have to be used to allow it to compile. HPACKIO and HPACKLIB: --------------------- All I/O functions are handled by the three libraries, HPACKIO (for filesystem I/O), HPACKLIB (for console I/O), and SYSTEM (for miscellaneous OS routines). These libraries contain code to interface HPACK with the underlying OS of the host system at the lowest level (HPACKIO corresponds to the system-level I/O functions, HPACKLIB corresponds to the stdio functions, and SYSTEM contains various nonstandard routines). The library routines are organised as follows: HPACKIO Library: hcreat(); - Create a given file hopen(); - Open a given file hclose(); - Close a given file hread(); - Read data from a file hwrite(); - Write data to a file hlseek(); - Move the file position indicator htell(); - Return current file position indicator htruncate(); - Truncate a file at the current file position hunlink(); - Remove a file hmkdir(); - Create a directory hrename(); - Rename a file hchmod(); - Set the file's attributes HPACKLIB Library: hputchar(); - Output a single char hputchars(); - As hputchar() but with stealth mode checking hputs(); - Output a string, adding CRLF at end with steath mode chk. hprintf(); - Output a formatted string hprintfs(); - As hprintf() but with stealth mode checking hgetch(); - Get a single char, no echo hmalloc(); - Allocate a block of memory hfree(); - Free a block of memory SYSTEM Library: setFileTime(); - Set the file's timestamp (non-standard) setDirTime(); - Set the directory's timestamp (non-standard) getCountry(); - Get country information for printing dates (non-standard) getScreenSize();- Get the screen size for user I/O (non-standard) isSameFile(); - Determine whether two pathnames refer to same file (nonstd) findFirst(); - Return first matching entry in directory (non-standard) findNext(); - Return following matching entries in directory (non-std) findEnd(); - End of findFirst/Next() routines (non-standard) copyExtraInfo();- Copy any extra information from one file to anoter(non-std) By default HPACKIO.H and HPACKLIB.H define these functions as the standard system-level I/O functions. However some of the functions (in particular those in SYSTEM.H) are nonstandard and will need to be supplied by the user (an implementation for Unix is contained in the file UNIX.C, an implementation for OS/2 is contained in the file OS2.C). The I/O functions often make use of file descriptors (given here as FD) to refer to files, and are as follows: int hcreat( const char *fileName, int mode ); Create file 'fileName' with access mode 'mode'. int hopen( const char *fileName, int mode ); Open file 'fileName' with access mode 'mode'. int hclose( const FD theFile ); Close file associated with 'theFile'. int hread( const FD theFile, const BYTE *buffer, const int count ); Reads 'count' bytes of data from the file associated with 'theFile' into buffer 'buffer'. int hwrite( const FD theFile, const BYTE *buffer, const int count ); Writes 'count' bytes of data to the file associated with 'theFile' from buffer 'buffer'. int hlseek( const FD theFile, const long offset, const int origin ); Moves the file position indicator for the file associated with 'theFile' to offset 'offset' from starting position 'origin'. long htell( const FD theFile ); Returns current file position indicator in file 'theFile'. int htruncate( const FD theFile ); Truncates the file associated with 'theFile' at the current file position. int hunlink( const char *fileName ); Removes file 'fileName'. int hmkdir( const char *dirName ); Creates directory 'dirName'. int hrename( const char *oldName, const char *newName ); Renames file 'oldName' to file 'newName'. int hchmod( const char *fileName, const int mode ); Set the file 'fileName's attributes to 'mode'. The HPACKLIB functions are as follows: int hputchar( const int ch ); Writes 'ch' to STDOUT. int hputchars( const char ch ); Writes 'ch' to STDOUT with stealth-mode checking. int hputs( const char *str ); Outputs string 'str' to STDOUT, adding CRLF at end with steath mode check. int hprintf( const char *format, ... ); Output a formatted string with format specified by 'format' to STDOUT. int hprintfs( const char *format, ... ); As hprintf() but with stealth mode checking. int hgetch( void ); Get a single character, no echo. This function is not entirely portable but is used in place of the standard getchar() since getchar() uses buffered I/O, making it possible for users to enter an arbitrary number of characters in response to a question which needs a yes/no answer. void hflush( FILE *stream ); Flush screen output buffers (required by Unix, and little else). void *hmalloc( const int size ); Allocates a block of memory of size 'size'. int hfree( void *memBlock ); Frees a block of memory pointed to by 'memBlock'. The SYSTEM functions are as follows: int setFileTime( const FD theFile, const long time ); Set the file 'theFile's timestamp to 'time' (non-standard). int setDirTime( const FD theDir, const long time ); Set the directory 'theDir's timestamp to 'time' (non-standard) int getCountry( void ); Get country information for printing dates (0 = US, 1 = European, 2 = Japanese format) (non-standard). void getScreenSize( void ); Get the screen dimensions and place them in the global variables screenHeight and screenWidth (non-standard). BOOLEAN isSameFile( const char *pathName1, const char *pathName2 ); Determine whether the two pathnames refer to the same file. MSDOS can do this by qualifying the names using an undocumented interrupt and comparing them; Unix can check the device numbers and inodes. This takes care of things like links, aliases, and so on (non-standard). BOOLEAN findFirst( const char *filePath, const ATTR matchAttr, \ const FILEHDR *fileInfo ); Returns first matching directory entry on 'filePath' (for example "a/b/c" would return "c" in directory "a/b" (if it existed)) with attributes matching 'matchAttr' and places info in 'fileInfo'. Returns TRUE if file found, FALSE otherwise (non-standard). BOOLEAN findNext( const FILEHDR *fileInfo ); Returns next matching directory entry and places info in 'fileInfo' (non- standard). void findEnd( const FILEHDR *fileInfo ); Cleanup function for findFirst()/findNext() calls, to be called after the last call to findNext() for a particular directory (non-standard). void copyExtraInfo( const FD srcFD, const FD destFD ); Copy any extra information (icons, extended attributes, etc) which are not part of the normal file data from one file to another (non-standard). HPACK Source Files: ------------------- The HPACK source files are as follows: arcdir.h - Interface and data structures for ARCDIR.C choice.h - The commands available for HPACK error.h - Interface for the error() routine errorlvl.h - Error level return codes filehdr.h - The file header structure filesys.h - interface for the filesystem handling routines flags.h - The switches available for HPACK frontend.h - Interface for FRONTEND.C system.h - System-specific information for OS interface routines tags.h - The master tags file for all versions of HPACK timeinfo.h - Time format header and conversion routines wildcard.h - Interface for the wildcard handling routines arcdir.c - Archive directory management routines arcdirio.c - Archive directory file I/O routines archive.c - The routines to store data in/retrieve data from an archive frontend.c - The archiver frontend, which contains most of archiver glue viewfile.c - The routine to display an archive directory cli.c - Generic CLI version frontend routines gui.c - Generic GUI version frontend routines error.c - The error() routine filesys.c - The filesystem handling routines script.c - The command-line and script filename handling routines tags.c - Tags handling code wildcard.c - Wildcard handling routines amiga.c - Amiga OS-specific routines arc.c - Archimedes OS-specific routines atari.c - Atari ST OS-specific routines mac.c - Macintosh OS-specific routines os2.c - OS/2 1.x OS-specific routines os2_32.c - OS/2 2.0 OS-specific routines unix.c - Unix OS-specific routines vms.c - VMS OS-specific routines crc/crc16.h - The CRC16 interface crc/crc16.c - Block CRC16 routines crypt/crypt.h - The encryption system header file crypt/crypt.c - The encryption management code crypt/md5.h - MD5 message digest header file crypt/md5.c - MD5 message digest routines crypt/nsea.h - Conventional-key encryption header file crypt/nsea.c - Conventional-key encryption routines crypt/rsa.h - The RSA library interface crypt/rsa.c - The RSA encryption library crypt/packet.h - Encryption packet definitions io/display.c - Formatted text display code io/fastio.h - Interface for the fast I/O routines io/fastio.c - The fast I/O system store/store.h - Interface to the store/unstore routines store/store.c - File store/unstore routines data/ebcdic.h - EBCDIC translation table For the LZW version: lzw/lzw.h - Information for the LZW compression/decompression routines lzw/lzw.c - LZW buffer allocation routines lzw/lzw_pack.c - LZW compression routine lzw/lzw_unpk.c - LZW decompression routine For the LZA' version: lza/lza.c - The LZA' main compressor/decompressor lza/pack.c - The arithmetic coder pack() routine lza/unpack.c - The arithmetic coder unpack() routine lza/model.h lza/model.c - The model for literals lza/model3.h lza/model3.c - The model for high positions lza/model4.h lza/model4.c - The model for low positions Multilingual support files: language/hpaktext.def - Text definitions file language/hpaktext.h - Interface for English (default) HPACK language/hpak_de.h - Interface for German HPACK language/hpak_nl.h - Interface for Dutch HPACK language/hpak_it.h - Interface for Italian HPACK language/hpaktext.c - English (default) text for HPACK language/hpak_de.c - German text for HPACK language/hpak_nl.c - Dutch text for HPACK langauge/hpak_it.c - Italian text for HPACK Supplementary files: makefile - Makefile for the Unix version makefile.os2 - Makefile for the OS/2 version docs/readme.1st - README file for HPACK docs/hpack.1 - HPACK manpage (nroff source for HPACK.DOC) docs/hpack.doc - The HPACK documentation docs/hpack.ps - The HPACK documentation in PostScript format docs/hpackext.doc - The HPACK extended documentation docs/hpackstd.txt - The HPACK standards document docs/hpaktest.txt - The HPACK torture test docs/hpaksmpl.txt - The HPACK sample archives (uuencoded) docs/hpakidea.txt - New ideas for HPACK data/stdarg.h - <stdarg.h> for those systems which don't have it data/stdlib.h - <stdlib.h> for those systems which don't have it data/winhpack.h - Windoze include file for HPACK data/winhpack.rc - Windoze resource file for HPACK data/hpack.pi.rsrc.hqx - BinHex'd Mac resource file for HPACK data/hpack.pi.hqx - BinHex'd ThinkC project file for HPACK data/testio.c - Test code for <os_name>.C filesystem I/O routines. This module can be linked with <os_name>.C to produce a standalone executable which will test most of the filesystem I/O calls without linking in the rest of HPACK. data/*.gif - Suggested screen / dialog layout for GUI versions (actually screen dumps from the Windows version of HPACK). The individual files are: data/addfiles.gif - Add files dialog data/authent.gif - Authentication options dialog data/encrypt.gif - Encryption options dialog data/extract.gif - Extract files dialog data/mainscrn.gif - Main HPACK screen data/miscopt.gif - Miscellaneous options dialog data/overwrit.gif - Overwrite options dialog data/translat.gif - Translate options dialog PGP 2.0 Keyring Maintenance: keycvt/makefile - Unix makefile for building the key conversion utility keycvt/keycvt.c - PGP 2.0 -> HPACK secret key format converter keycvt/idea.c - Cipher for decrypting PGP 2.0 secret keys keycvt/md5.h keycvt/md5.c - Message digest code needed by keycvt keycvt/mdc.h keycvt/mdc.c - Cipher for encrypting HPACK secret keys Coding: ------- The following notes outline some of the coding conventions used in HPACK. - System-specific code: The system being used should be specified on the command-line (generally in the makefile) by a define of the form __<system>__, for example __MSDOS__ or __UNIX__. For some systems extra defines may be necessary, for example Unix systems may require the extra defines BSD, SYSV, IRIX, ULTRIX, POSIX, etc. Any system-specific code should be enabled/disabled through these defines. As an example, to build HPACK under Ultrix, the command-line options for the compiler would include '-D__UNIX__' and '-DULTRIX' (again these are given in the makefile). - General assumptions about types: The following types are defined in DEFS.H typedef <8-bit unsigned value> BYTE typedef <16-bit unsigned value> WORD typedef <32-bit unsigned value> LONG typedef <64-bit unsigned value> QUAD typedef <128-bit unsigned value> LQUAD When used in HPACK archives, all data are stored in big-endian format, ie for a WORD the byte ordering is [MSB:LSB]. The get/put byte/word/long/ quad/lquad functions perform automatic endianness conversion. The choice of endianness is largely irrelevant because of this automatic conversion; the final decision was made based on the fact that various Internet and US Government encryption standards require data to be big-endian. A BOOLEAN is any value capable of holding a value of TRUE or FALSE (1 or 0). The standard HPACK distribution uses unsigned chars simply because these take up the least space. - Endianness and packing/non-packing of structs: The get/put byte/word/long/quad/lquad routines perform automatic endianness conversion as data is read/written. The field-by-field reading and writing of structs avoids any problems with compilers which align all struct fields to word or longword boundaries. - Bitfields: The ANSI standard for bitfields states that they are implementation- dependant. HPACK implements bitfields using bitmasks to avoid this problem. - Coding style: Try and keep the coding style identical to that used throught the rest of HPACK. I will probably go through and change any submitted code to conform to this coding style in order to preserve a unified style across all source files. At least one reason for this is that the automatic format conversion program 'mangle' expects code to be laid out in a certain way and will probably break if it hits an unexpected code layout. - Making changes to HPACK code: In general the main HPACK code should be modified as little as possible. All changes should be restricted to <os_name>.C and SYSTEM.H. Again, I will nitpick any submitted code and recommend moving things into <os_name>.C if necessary. The reason behind this is that too much code which runs over multiple systems consists of an indecipherable morass of #ifdef'd code blocks for every system it has ever been compiled on. The purpose of the 'h'-functions, <os_name>.C, and SYSTEM.H, is to keep this to an absolute minimum. Releasing an HPACK Port: ------------------------ In order to ensure complete consistency across all versions of HPACK, several points should be observed: - The file TAGS.H should *never* be modified. If any changes are needed, they should be submitted to me, and I will make the changes if necessary and redistribute the new TAGS.H to all HPACK coders. This is necessary to ensure all parts of HPACK archives are understandable by all other versions of HPACK (as far as this is possible). - There is a file HPAKTEST.TXT included in this distribution which gives a checklist of features in the basic version of HPACK (this is also known as "The HPACK Torture Test"). Only ports which *completely* pass the test (as far as is applicable) should be released as final releases. Non-complying versions should be labelled alpha-, beta-, or development releases as appropriate (I waited over 2 years before releasing the first versions of HPACK to make sure I'd Got It Right the first time). HPACK on non-ANSI Systems: -------------------------- Some systems have strange non-ANSI compilers (which nevertheless claim to be ANSI - some of them (eg Nyarlathotep, the ESIX compiler) are more like the missing link than a C compiler). To convert HPACK for these compilers, the program 'mangle' is included as DATA/MANGLE.C. Mangle can transform the source in a variety of bizarre and unusual ways, and takes as input a list of options telling it what to do with the code, and a filename on which to perform the transformations (NB some of the mangling options may not be legal in some countries - use at your own risk). Options include: -c - Get rid of '\' line continuation characters (except in macros and strings). -e - Change all occurrences of #elif to #else #if ... #endif -f - Turn function headers into K&R versions, for example int getFileID( const char *fileName, int dirID ) becomes int getFileID( fileName, dirID ) char *fileName; int dirID; -i - Unindent all lines beginning with '#' (some preprocessors expect directives to always begin in column 1). -k - Delete 'const' from the code. -n - Turn enumerations into #defines and integer types. -p - Change all occurrences of 'void *' to 'char *'. -r - Delete '#pragma' lines from the code. -s - Concatenate strings split over multiple lines with '\'. -t - Turn function prototypes into nameless ANSI versions, for example int getFileID( const char *fileName, int dirID ); becomes int getFileID( const char *, int ); -v - Delete all 'void's in the code (return values become 'int', parameters disappear). -w - Turn function prototypes into K&R versions, for example int getFileID( const char *fileName, int dirID ); becomes int getFileID(); Mangle is written in ANSI C so it may be necessary to find another system on which to cross-mangle the code before compiling it on the destination system. Note that mangle expects the code to be formatted in a certain way (as it always is in HPACK). It doesn't appear to break any code even when subject to maximum mangling, but I gave up checking the code after the 300th warning message from a genuine ANSI compiler (these warnings were mainly due to the destruction of function prototypes). At least one compiler, if told to #include "a/b", will try to include "a/a/b". The compiler owner tells me all SYSV compilers do this, in which case I extend my condolences. If your system doesn't have a <stdlib.h>, there is one in the directory 'data' which can be moved into the main HPACK directory. If your system doesn't have a <stdarg.h>, there is also one in the 'data' directory. When compiling error.c, define NO_STDARG to include different code which doesn't assume ANSI va_arg functions. This code assumes all data is passed on the stack, and may have to be altered for RISC machines which try to pass as much as possible in registers. Who's Doing What: ----------------- The following is a list of people currently claiming to be working on HPACK and their email contact addresses and phone numbers: HPACK/DOS Peter Gutmann - pgut1@cs.aukuni.ac.nz (preferred) peter@nacjack.gen.nz or peterg@kcbbs.gen.nz Ph.(09) 426-5097 HPACK/Windoze Lynn Prentice - lprent@kcbbs.gen.nz HPACK/Unix Stuart Woolford - stuartw@ccu1.aukuni.ac.nz Ph.(09) 426-3464 HPACK/OS2 John Burnell - johnb@maths.grace.cri.nz HPACK/IIGS Corey Murtagh - Corey_Murtagh@kcbbs.gen.nz Ph.(09) 277-5800 HPACK/Mac Peter Gutmann - pgut1@cs.aukuni.ac.nz (preferred) peter@nacjack.gen.nz or peterg@kcbbs.gen.nz Ph.(09) 426-5097 Bonus Info: HPACK Character Sheet: ----------------------------------- HPACK (Archiver Supremus Maximus) FREQUENCY: 8 (Amiga, Archimedes, Atari ST, Macintosh, MSDOS, Windoze, OS/2, Unix) NO. APPEARING: Not a lot ARMOUR CLASS: 8 (2 on an OS with protection) MOVE: Bloody slow HIT DICE: Constantly % IN LAIR: 100 TREASURE TYPE: Easter eggs (many) NO. OF ATTACKS: Unlimited DAMAGE/ATTACK: May affect sanity of users. Has been known to destroy hardware - see "The HPACK Curse" in README.1ST. SPECIAL ATTACKS: Running HPACK may be classed as a denial of service-type attack by some systems administrators. SPECIAL DEFENSES: Mutates constantly to confuse foes MAGIC RESISTANCE: Immune to interrupts of all kinds (there are rumours of variants of HPACK which will even survive system resets). INTELLIGENCE: Just enough to be considered alive. ALIGNMENT: Chaotic evil SIZE: S (but growing) PSIONICS: Psionics? How do you spell that?